Repository navigation
Antalya 26.8: Fix race condition in Glue update - #2861
Conversation
…e-can-drop-a-commit-that-lands-while-it-is-publishing
Use Glue VersionId for compare-and-swap on UpdateTable to prevent OPTIMIZE from silently overwriting a concurrent commit (e.g. INSERT) that lands while the compaction is publishing. Changes: - GlueCatalog::getTableVersionId: new helper that fetches the current VersionId via GetTable before each update. - GlueCatalog::updateTableInGlue: accepts an optional version_id; when set, passes it to request.SetVersionId so Glue rejects the update on version mismatch (CONCURRENT_MODIFICATION / VERSION_MISMATCH). Returns false on conflict instead of throwing, matching the REST catalog convention that callers already handle. - GlueCatalog::updateMetadata / updateSchema: fetch VersionId before calling updateTableInGlue. - BinPackRewrite: move the iceberg_compaction_pause_before_metadata_commit failpoint before writeMetadataToStorage so a concurrent writer can complete its full commit (S3 file + Glue pointer) during the pause. - Remove @pytest.mark.xfail from test_optimize_bin_pack_concurrent_insert; all three rows now survive. Closes: #2733
…-that-lands-while-it-is-publishing' of https://github.com/Altinity/ClickHouse into 2733-optimize-on-a-glue-iceberg-table-can-drop-a-commit-that-lands-while-it-is-publishing
CI triage for c28ec19 (CI still running)So far 1 check has failed,
The targeted job did not pick these tests from this PR's diff. Its job log lists exactly these 3 tests ( 1.
|
…into 2733-optimize-on-a-glue-iceberg-table-can-drop-a-commit-that-lands-while-it-is-publishing
CI triage for
|
| Check | Failure | Classification |
|---|---|---|
| Integration tests (amd_asan_ubsan, targeted) and (… db disk, old analyzer, 7/8) | test_backward_compatibility/test_custom_s3_disk_remote_url_allow_hosts.py (both tests) |
Pre-existing on base |
| Stateless tests (amd_asan_ubsan, cas s3 storage, parallel, 1/2 and 2/2) | Job timeout | Pre-existing / infra |
| Stateless tests (amd_binary, cas s3 storage, parallel) | 05055_format_schema_source_query_background_access |
Test bug (timezone + midnight), unrelated |
Regression release settings, swarms, s3_azure_1 |
Suite failures | Pre-existing on base |
Regression release cas_s3_cache_lightweight_delete_4 |
/lightweight delete/concurrent delete/MergeTree/random delete entire table |
Unrelated code path, likely flaky; re-run |
1. test_custom_s3_disk_remote_url_allow_hosts.py: pre-existing
The cluster never starts, so both tests error during setup:
E Exception: Timed out while waiting for instance `node' with ip address 172.16.1.7 to start. Container status: running
The same two tests fail the same way on other antalya-26.8 PRs: #2873 (db disk 7/8), #2878 (targeted), and #2881 (targeted and 7/8). The test came in with the backport 6f08453ad8e ("Fix ATTACH for S3 disks created without 'remote_url_allow_hosts'"). The "targeted" job picks it up because it was changed recently on the base branch, not because of this PR. It needs a separate fix on antalya-26.8.
2. Stateless ASan "cas s3 storage" 1/2 and 2/2: pre-existing / infra
No test failed. The shards were killed while still running tests (shard 1 had reached 5146 of 6890 tests):
WARNING: Job timed out: [Stateless tests (amd_asan_ubsan, cas s3 storage, parallel, 1/2)], timeout [12600], exit code [-15]
Both shards hit the same ERROR on #2873 and #2881. The job is too slow for its 3.5h budget on this branch.
3. 05055_format_schema_source_query_background_access: test bug, unrelated
-background flush refused: 1
+background flush refused: 0
The CI's randomized-settings diagnosis found that the test passes 17/17 without randomization, and one of the culprit settings it named was --session_timezone Africa/Khartoum. The test sets START=$(… SELECT toString(now64(6))) in the session timezone (2026-11-17 00:00:55), then filters event_date >= toDate('${START}'). event_date is in the server timezone, where it was still 2026-11-16 (the runner was at −05:00). Near midnight that filter drops every row. The test only reaches this PR because it was backported in ef62ce8ec19, and it doesn't touch Glue or Iceberg.
Fix (separate PR to antalya-26.8, or upstream): drop the event_date predicate, or capture START as a timezone-independent value, e.g. SELECT toUnixTimestamp64Micro(now64(6)), and compare with toUnixTimestamp64Micro(event_time_microseconds) >= ${START}.
4. Regression settings / swarms / s3_azure_1: pre-existing
These three also fail on #2881 (settings, swarms, s3_azure_1), and swarms fails on #2873 as well. settings reports 1859 of 1865 scenarios failed in under 2 minutes, which is a broken suite or snapshot, not a product change.
5. Regression cas_s3_cache_lightweight_delete_4: unrelated, likely flaky
✘ [ Fail ] '/lightweight delete/concurrent delete/MergeTree/random delete entire table' (1m 3s)
File ".../lightweight_delete/tests/concurrent_delete.py", line 285, in random_delete_percentage_of_the_table
AssertionError: Oops! Assertion failed
This is a concurrent lightweight-DELETE scenario on plain MergeTree. The PR only changes GlueCatalog.{h,cpp}, BinPackRewrite.cpp (Iceberg OPTIMIZE) and the Glue integration test, none of which run in this scenario. The job passed on #2881 and #2873. I haven't seen this exact failure elsewhere, so this isn't proven flaky. Re-run it; if it fails again, it is worth a separate look.
Suggested next step: nothing to change in this PR. Re-run the cas_s3_cache_lightweight_delete_4 regression job to confirm. The backward-compat test, the stateless ASan "cas s3" timeout and 05055 need fixes on antalya-26.8 itself; I can open a PR for the 05055 fix if you want.
Evidence: praktika result_pr.json for this SHA and for #2873, #2878, #2881; the integration shard job.log files; the GH Actions log of job 133687578907.
mkmkme
left a comment
There was a problem hiding this comment.
🤖 [Posted by an LLM agent on behalf of @mkmkme]
The VersionId is read right before UpdateTable, after any concurrent commit has already landed, so this doesn't close the race from #2733. The new integration test passes without the GlueCatalog change. See the inline comments.
Also, the comment in commitMetadata (BinPackRewrite.cpp:1343-1345) still says Glue UpdateTable "is unconditional" and that the check "cannot close" the window. Please update it to match the final design.
Changelog entry: "an optimize call can overwrite a commit in process" is vague. Suggested: "Fix OPTIMIZE TABLE and other Iceberg writes to a Glue catalog table silently discarding a commit made by another writer while the operation was publishing its metadata."
| bool GlueCatalog::updateMetadata(const String & namespace_name, const String & table_name, const String & new_metadata_path, Poco::JSON::Object::Ptr /*new_snapshot*/, const DB::ForwardedAuthTokenPtr & auth_token) const | ||
| { | ||
| return updateTableInGlue(namespace_name, table_name, new_metadata_path, auth_token); | ||
| const auto version_id = getTableVersionId(namespace_name, table_name, auth_token); |
There was a problem hiding this comment.
🤖 [Posted by an LLM agent on behalf of @mkmkme]
This reads the VersionId too late. It describes the table after any concurrent commit has landed, so the update succeeds anyway.
For example: OPTIMIZE builds on v5. Spark commits v6-<uuid> and Glue's version becomes 7. OPTIMIZE then reads version 7 here and sends UpdateTable(..., VersionId=7). Glue accepts it, and Spark's commit is lost.
Iceberg Java's GlueTableOperations.doCommit reads the table with GetTable, checks that its metadata_location still equals the writer's base (checkMetadataLocation), and only then sends UpdateTable with that table's versionId(). The location check is what's missing here.
updateMetadata needs to know the expected base. That could be a new argument on ICatalog::updateMetadata, or a comparison of the new snapshot's parent-snapshot-id with Glue's current snapshot. Every other updateMetadata caller (INSERT, mutations, drop partition, expire snapshots, manifest compaction) would get the same fix.
There was a problem hiding this comment.
Updated: VersionId is now read in the same GetTable`call
| if (new_schema) | ||
| columns = icebergSchemaToGlueColumns(new_schema); | ||
| return updateTableInGlue(namespace_name, table_name, new_metadata_path, auth_token, columns); | ||
| const auto version_id = getTableVersionId(namespace_name, table_name, auth_token); |
There was a problem hiding this comment.
🤖 [Posted by an LLM agent on behalf of @mkmkme]
Same problem as in updateMetadata above.
|
|
||
| LOG_WARNING(getLogger("GlueCatalog"), | ||
| "Failed to fetch VersionId for {}.{}: {}", namespace_name, table_name, outcome.GetError().GetMessage()); | ||
| return {}; |
There was a problem hiding this comment.
🤖 [Posted by an LLM agent on behalf of @mkmkme]
When GetTable fails, this silently disables the protection: UpdateTable then goes out without a version, which is the old unconditional overwrite. The other GetTable callers in this file throw DATALAKE_DATABASE_ERROR on errors other than not-found, and this one should too.
Reproducing gtest: https://gist.github.com/mkmkme/c50d201e1700ea48367759ed783f05b8. GetTableFailureDoesNotSendUnconditionalUpdate fails on this PR and passes once this path throws.
There was a problem hiding this comment.
Added getTableVersion, it throws DATALAKE_DATABASE_ERORR when GetTable fails.
|
|
||
| # 3. Session B from the issue: a commit lands in the publish window. | ||
| node.query( | ||
| f"INSERT INTO {table_ref} VALUES ('inserted-during-optimize');", |
There was a problem hiding this comment.
🤖 [Posted by an LLM agent on behalf of @mkmkme]
This test passes without the GlueCatalog change; I ran it on antalya-26.8 with only the failpoint hook added. Glue isn't a transactional catalog, so this INSERT writes v4.metadata.json. OPTIMIZE then tries to write v4.metadata.json as well, hits Bin-pack commit conflict detected, cleaning up in writeMetadataToStorage, and retries without ever reaching updateMetadata.
The case that can lose data is a writer that names its metadata files differently, such as Spark, Athena or pyiceberg. Committing through pyiceberg (load_catalog_impl) at this point would exercise the fix. That works with moto once the location check is in place. Moto ignores VersionId on UpdateTable, both in 5.1.5 and the current 5.2.5, so the CONCURRENT_MODIFICATION path can't be reached from integration tests at all.
There was a problem hiding this comment.
test_database_glue/test.py::test_listing_propagates_non_not_found_error
[gw0] [ 92%] PASSED test_database_glue/test.py::test_listing_propagates_non_not_found_error
test_database_glue/test.py::test_exists_table_answers_only_from_not_found
[gw0] [ 94%] PASSED test_database_glue/test.py::test_exists_table_answers_only_from_not_found
test_database_glue/test.py::test_catalog_schema_with_empty_column_name_is_rejected
[gw0] [ 97%] PASSED test_database_glue/test.py::test_catalog_schema_with_empty_column_name_is_rejected
test_database_glue/test.py::test_optimize_bin_pack_concurrent_insert
[gw0] [100%] PASSED test_database_glue/test.py::test_optimize_bin_pack_concurrent_insert
=============================== warnings summary ===============================
test_database_glue/test.py: 270 warnings
/root/Documents/ClickHouse/tests/integration/test_database_glue/test.py:159: DeprecationWarning: datetime.datetime.utcnow() is deprecated and scheduled for removal in a future version. Use timezone-aware objects to represent datetimes in UTC: datetime.datetime.now(datetime.UTC).
datetimes.append(datetime.utcnow() - timedelta(minutes=random.randint(0, 60)))
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
- generated report log file: /root/Documents/ClickHouse/ci/tmp/pytest_parallel.jsonl -
================= 38 passed, 270 warnings in 447.42s (0:07:27) =================
Run command: [sudo iptables -D DOCKER-USER 1 ||:]
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Ok [Integration tests (amd_tsan, 1/8)]
| Failures: 0/38
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Audit review: PR #2861 — Fix race condition in Glue updatePR: #2861 (head Related: #2733 Confirmed defects1. Medium — A rejected Glue commit leaves a metadata file in storage that points to a deleted manifest listDescription Before this PR, Glue is not a transactional catalog (
// IcebergWrites.cpp, IcebergStorageSink::initializeMetadata
if (!writeMetadataFileAndVersionHint(..., metadata_info, ..., hint_path, ...)) // vN+1.metadata.json written
...
if (!catalog->updateMetadata(...)) // can now return false for Glue
{
cleanup(true); // removes the manifests and the manifest list, not metadata_info.path or the hint
return false;
}ClickHouse's own reads through the Glue database follow the catalog pointer, so they are not affected. Path-based readers of the same table are: The window is small today, because Steps to reproduce (real AWS Glue is needed: moto ignores
|
|
The fix compares
1. Start LocalStack {#start-localstack}docker run -d --name localstack -p 4566:4566 \
-e SERVICES=glue,s3 -e LOCALSTACK_AUTH_TOKEN=$LOCALSTACK_AUTH_TOKEN \
localstack/localstack-pro # or localstack/localstack if Glue works on your account
export AWS_ACCESS_KEY_ID=test AWS_SECRET_ACCESS_KEY=test AWS_DEFAULT_REGION=us-east-1
python3 -m venv tmp/venv && . tmp/venv/bin/activate
pip install awscli-local 'pyiceberg[glue,pyarrow]'
awslocal s3 mb s3://warehouse-glue2. Start ClickHouse {#start-clickhouse}Use a build that includes the fix (the mkdir -p tmp/ch && cd tmp/ch
../../build/programs/clickhouse server # terminal 1
../../build/programs/clickhouse client # terminals 2 and 3 (clients A and B)3. Create the table with pyiceberg {#create-table}
import pyarrow as pa
from pyiceberg.catalog import load_catalog
from pyiceberg.schema import Schema
from pyiceberg.types import NestedField, LongType
cat = load_catalog("test", **{
"type": "glue",
"glue.endpoint": "http://localhost:4566",
"glue.region": "us-east-1",
"s3.endpoint": "http://localhost:4566",
"s3.access-key-id": "test",
"s3.secret-access-key": "test",
})
cat.create_namespace("ns")
t = cat.create_table(
"ns.t",
Schema(NestedField(1, "x", LongType(), required=False)),
location="s3://warehouse-glue/t",
)
# Two appends give two small data files, so bin-pack has work to do.
t.append(pa.table({"x": pa.array([1, 2], pa.int64())}))
t.append(pa.table({"x": pa.array([3, 4, 5], pa.int64())}))python tmp/setup.py4. Attach the Glue database in ClickHouse (client A) {#attach-database}SET allow_database_glue_catalog = 1, write_full_path_in_iceberg_metadata = 1;
CREATE DATABASE g ENGINE = DataLakeCatalog('http://localhost:4566', 'test', 'test')
SETTINGS catalog_type = 'glue', warehouse = 'test', region = 'us-east-1',
storage_endpoint = 'http://localhost:4566/warehouse-glue',
aws_access_key_id = 'test', aws_secret_access_key = 'test';
SELECT count() FROM g.`ns.t`; -- expect 55. Reproduce the race {#reproduce-race}Client A: enable the failpoint and start SYSTEM ENABLE FAILPOINT iceberg_compaction_pause_before_metadata_commit;
SET allow_experimental_iceberg_compaction = 1,
iceberg_target_data_file_size_bytes = 10485760,
iceberg_min_data_file_size_bytes = 10485760;
OPTIMIZE TABLE g.`ns.t`;Client B: wait until SYSTEM WAIT FAILPOINT iceberg_compaction_pause_before_metadata_commit PAUSE;Shell: land a concurrent commit through pyiceberg while import pyarrow as pa
from pyiceberg.catalog import load_catalog
cat = load_catalog("test", **{
"type": "glue",
"glue.endpoint": "http://localhost:4566",
"glue.region": "us-east-1",
"s3.endpoint": "http://localhost:4566",
"s3.access-key-id": "test",
"s3.secret-access-key": "test",
})
cat.load_table("ns.t").append(pa.table({"x": pa.array([6, 7, 8, 9], pa.int64())}))python tmp/concurrent.pyClient B: release the failpoint. Client A's SYSTEM DISABLE FAILPOINT iceberg_compaction_pause_before_metadata_commit;6. Check the result {#check-result}SELECT count() FROM g.`ns.t`;or cat.load_table("ns.t").scan().to_arrow().num_rows
|
…tables that dont have metadata_location parameter
closes #2733
Changelog category (leave one):
Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):
Fix
OPTIMIZE TABLEon an Iceberg table in a Glue catalog silently discarding a commit that another writer made while the compaction was running. The compaction now publishes its result only if Glue still points at the metadata it started from, and otherwise retries on the new snapshot.Exclude tests:
Regression jobs to run: